fix: precompute project group merge suggestion counts (CM-1230) by skwowet · Pull Request #4250 · linuxfoundation/crowd.dev

skwowet · 2026-06-22T20:14:39Z

Summary

CM-1230 addressed membersToMerge / organizationsToMerge timeouts by scoping merge suggestions against rollup rows in memberSegmentsAgg / organizationSegmentsAgg — matching the selected project-group segment directly instead of expanding to leaf subprojects and hitting the agg tables with huge segmentId IN (...) lists.

That fixed the worst leaf-expansion cases, but unfiltered COUNT requests were still slow for large project groups: even with a single rollup segment ID, Postgres still scanned all merge rows and ran expensive EXISTS checks against very large agg buckets, which could exceed API timeouts.

This PR adds precomputed per–project-group merge suggestion counts (same idea as other agg-backed reads: expensive work offline, cheap read online). Unfiltered badge/list counts are served from segmentMergeSuggestionCounts; filtered counts still use the live query — that is intentional and fine: the production incident was unfiltered countOnly on huge project groups (e.g. LF OSS Index), not filtered data-quality views. Filtered requests (similarity, displayName, memberId/organizationId) keep the existing live COUNT path; they were not timing out before this PR and do not need precomputation.

Also restores project / subproject filtering on the merge suggestions list page (Projects dropdown). Segment scoping had regressed to project-group-only; list and live COUNT again use filter.projectIds / filter.subprojectIds when set, with precomputed counts only for unfiltered project-group scope.

Counts are refreshed on a schedule and adjusted immediately when users merge or dismiss suggestions.

How it works now

Cron (every 4h) computes member and org merge suggestion counts per active project group and upserts segmentMergeSuggestionCounts (skips write when values are unchanged).
API COUNT (unfiltered) — project group segment (parent/grandparent null), no similarity / display name / entity filters → read precomputed row (~ms). CDP UI always sends the selected project group as the sole segment.
API COUNT (filtered) — any filter present → existing live COUNT against memberToMerge / organizationToMerge + agg EXISTS. Same query shape as before this PR; acceptable at query time because filtered views are not the high-volume unfiltered badge path and were not the timeout source.
LIST — still paginated live queries; respects project/subproject filters when provided. Only the count side uses the table when unfiltered at project-group scope.
User actions — on member/org merge or no-merge, decrement the precomputed count for project groups where both entities share a rollup segment, so list badges update without waiting for cron.

Changes

Migration: segmentMergeSuggestionCounts table (segmentId, member/org counts, updatedAt).
DAL (segments/index.ts): fetch project groups; calculate/upsert/get counts; common project-group segment lookup; decrement helpers.
Cron: refresh-segment-merge-suggestion-counts job (batched parallelism, fail-fast on segment errors).
Repositories: hybrid getTotalCount() — precomputed vs live; aligned member/org list + COUNT SQL (NOT EXISTS for mergeActions, consistent EXISTS); project-group guard for cache reads; simplified org list query (removed CTE / hash dedup).
Merge suggestions filtering: restore project/subproject segment filters on list + live COUNT (projectIds / subprojectIds); precomputed table remains project-group only when no project filter and no count filters.
Services: decrement precomputed counts after member merge, member no-merge, org merge, and org no-merge.

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot

Pull request overview

This PR introduces a precomputed, per–project-group cache of merge-suggestion counts to avoid expensive COUNT(*) queries timing out on large project groups, while keeping filtered counts and list pagination as live queries. It adds a cron refresher plus immediate count adjustments when users merge or dismiss suggestions.

Changes:

Adds segmentMergeSuggestionCounts table + DAL helpers to compute, upsert, fetch, and decrement counts per project-group segment.
Adds a cron job to refresh cached counts every 4 hours (only writing when counts change).
Updates member/org merge-suggestion repositories and merge/no-merge flows to use cached totals for unfiltered single-project-group counts and decrement counts on user actions.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
services/libs/data-access-layer/src/segments/index.ts	Adds DAL functions for project-group segment enumeration, count calculation, cached count upsert/get, common-segment lookup, and decrement helpers.
services/libs/common_services/src/services/common.member.service.ts	Decrements cached member merge-suggestion counts after a successful member merge.
services/apps/cron_service/src/jobs/refreshSegmentMergeSuggestionCounts.job.ts	New cron job that refreshes cached counts on a 4h schedule.
backend/src/services/organizationService.ts	Decrements cached org merge-suggestion counts after org merge and org no-merge actions.
backend/src/services/memberService.ts	Decrements cached member merge-suggestion counts after member no-merge actions.
backend/src/database/repositories/organizationRepository.ts	Uses cached totals for unfiltered single-project-group counts; adjusts mergeActions filtering in list/count SQL.
backend/src/database/repositories/memberRepository.ts	Uses cached totals for unfiltered single-project-group counts; adjusts mergeActions filtering in list/count SQL.
backend/src/database/migrations/V1782128525__segment-merge-suggestion-counts.sql	Adds the `segmentMergeSuggestionCounts` table schema.

Comments suppressed due to low confidence (1)

backend/src/database/repositories/organizationRepository.ts:820

Using a mergeActions LEFT JOIN + (ma.id IS NULL OR ma.state = ERROR) can multiply rows and can incorrectly include a suggestion if there are mergeActions recorded in both directions with different states (the ERROR row still passes). A NOT EXISTS predicate avoids join-multiplication and correctly excludes suggestions when any non-ERROR mergeAction exists.

      LEFT JOIN "mergeActions" ma
        ON ma.type = :mergeActionType
        AND (
          (ma."primaryId" = otm."organizationId" AND ma."secondaryId" = otm."toMergeId")
          OR (ma."primaryId" = otm."toMergeId" AND ma."secondaryId" = otm."organizationId")

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.

…ounts table Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

…ries to use single segment ID Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.

…y query Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

ulemons · 2026-06-24T07:37:03Z

@skwowet looks good to me, and this can seriously speed up things! ✅
Just one doubt, the job is already running in prod ? otherwise the first time we will have weird results right ? until at least a first round of count is computed

skwowet · 2026-06-24T07:46:06Z

@ulemons Good point. No, the job isn't running in prod yet.

I think the safest rollout would be:

Run the db migrations.
Deploy the cron job and let it populate the counts table.
Deploy the api changes once the initial counts have been computed.

That should avoid any weird results during the initial backfill

skwowet added 3 commits June 23, 2026 01:04

fix: precompute project group merge suggestion counts

a5907a0

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

fix: return count as a number instead of a string in member repository

f06a3be

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

revert: member and organization count filters

8958312

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

skwowet self-assigned this Jun 22, 2026

Copilot AI review requested due to automatic review settings June 22, 2026 20:14

Merge branch 'main' into improve/CM-1230

cc226af

Copilot started reviewing on behalf of skwowet June 22, 2026 20:15 View session

skwowet marked this pull request as ready for review June 22, 2026 20:16

fix: resolve pr review comments

f20dea8

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot AI reviewed Jun 22, 2026

View reviewed changes

skwowet added 4 commits June 23, 2026 02:10

fix: decrement calls and trnx usage

22cc2e9

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

fix: resolve pr review comments

ea0d0ad

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

fix: resolve pr review comments

c387a88

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Merge branch 'main' into improve/CM-1230

c04b63c

skwowet requested review from Copilot and ulemons June 23, 2026 07:08

Copilot started reviewing on behalf of skwowet June 23, 2026 07:08 View session

Copilot AI reviewed Jun 23, 2026

View reviewed changes

skwowet added 3 commits June 23, 2026 13:44

fix: add default value to updatedAt column in segmentMergeSuggestionC…

c471ece

…ounts table Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

refactor: update segment handling in member and organization reposito…

bff28b2

…ries to use single segment ID Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

fix: restore project and subproject filtering for merge suggestions

245b9a0

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 23, 2026 09:16

Copilot started reviewing on behalf of skwowet June 23, 2026 09:16 View session

Copilot AI reviewed Jun 23, 2026

View reviewed changes

fix: remove unnecessary null check for similarity in member repositor…

2fc787c

…y query Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

ulemons approved these changes Jun 24, 2026

View reviewed changes

Conversation

skwowet commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How it works now

Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ulemons commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

skwowet commented Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

skwowet commented Jun 22, 2026 •

edited

Loading

ulemons commented Jun 24, 2026 •

edited

Loading